Capturing the variability of internet flows in a workload generator for network simulators

The ever growing amount of transferred data and the constantly increasing expectance on reliability and performance in the Internet pose a fundamental problem for network designers. Traditionally these problems have been tackled by overprovisioning link and router capacity to ensure enough spare room. More recently however careful steering of traffic flows has been deployed as a more sophisticated solution to this problem. This so called Traffic Engineering is mostly done by manually configuration of network components to optimally distribute traffic demands over the available resources. There is also ongoing research on automated traffic management using load adaptive routing algorithms to reduce manual intervention on the network. Both approaches rely on the observation, that the size of traffic flows follows Zipf’s Law. This implies that only a small number of flows are responsible for the vast majority of the overall network traffic. Traffic Engineering as well as load adaptive routing try to exploit this property by identifying a small set of traffic flows and, by treating this small set in a special way, to control the flow of data in a network. Unfortunately the high variability of Internet traffic leads to the problem that the set of these few large traffic flows is not constant over time. Some of the traffic flows will cease to exist, new ones appear and existing ones change their size drastically over time. The consequence of this variability is the need for periodic reclassification and intervention into the network configuration. For load adaptive routing this implies the threat of instability through route oscillation. In this work we investigate the nature as well as the causes of the volatility of Internet flows and their impact on traffic engineering and load adaptive routing. We base our study on packet level and NetFlow traces, covering access networks and backbones of scientific as well as commercial networks and propose a methodology for studying persistence aspects of large Internet flows. We show the feasibility of using coarse grained NetFlow data to perform this kind of analysis. Using this methodology we study persistency aspects of large flows on multiple time resolutions and flow aggregations. Our analysis shows a significant stability of flows with high average flow rates, but that there is a high amount of variability in the set of the largest flows. We then use fine grained packet level traces to understand these persistency aspects. We identify the arrival of new and the departure of active flows and rate fluctuations for aggregated flows as the two main contributors of the observed inpersistency. We then introduce a methodology to automatically derive network and protocol parameters from traces of tightly controlled Web page retrievals in order to configure our simulations and show that the base components of our simulation environment, the ns-2 network simulator and the NSWeb traffic generator, are able to provide us with the needed accuracy and configurability. We then introduce our simulation environment and demonstrate, that we are able to reproduce the persistency properties and the variability of large network flows for the evaluation of traffic management approaches in simulations.